Reducing Confusion in Active Learning for Part-Of-Speech Tagging

Active learning (AL) uses a data selection algorithm to select useful training samples minimize annotation cost. This is now an essential tool for building low-resource syntactic analyzers such as part-of-speech (POS) taggers. Existing AL heuristics are generally designed on the principle of selecting uncertain yet representative instances, where annotating these instances may reduce large number errors. However, in empirical study across six typologically diverse languages (German, Swedish, Galician, North Sami, Persian, and Ukrainian), we found surprising result that even oracle scenario know true uncertainty predictions, current far from optimal. Based this analysis, pose problem maximally confusion between particular pairs output tags. Extensive experimentation aforementioned shows our proposed strategy outperforms other strategies by significant margin. We also present auxiliary results demonstrating importance proper calibration models, which ensure through cross-view training, analysis how selects examples more closely follow distribution. The code publicly released here. 1